Intelligent use of scene and test pattern analyses for traffic camera diagnostics

ABSTRACT

A method for determining a response to misalignment of a camera monitoring a desired area includes acquiring temporal related frames from the camera including a reference frame. A pixel location is determined of a reference object from the frames. Using the pixel location of the reference object, a displacement of the camera between a current frame and the reference frame is determined. For the displacement exceeding a first threshold, a new displacement of the camera is measured by introducing at least one additional object to a camera field of view and comparing the new displacement to a second threshold. For the new displacement not exceeding the second threshold, the camera is recalibrated using a determined pixel location and a physical location of the at least one additional object. For the new displacement exceeding the second threshold, notification is provided of a misalignment to an associated user device.

BACKGROUND

The present disclosure relates to a system and method for performingcamera diagnostics. The system operates using a multiple phase approachto detect and confirm camera deterioration. The present disclosure findsapplication in traffic surveillance. However, it is appreciated that thepresent exemplary embodiments are also amendable to other likeapplications.

Conventional traffic cameras are used to monitor traffic flow and surveyvehicle compliance with traffic laws, rules, and regulations. Aconventional traffic camera can support a variety of surveillance tasks.Generally, this type of camera selectively captures video of a scenethat is being surveyed. These cameras may then transmit image data to aremote server device for further processing, or may perform some or allof the processing on-board. Because the image data is often used for feecollection and traffic enforcement, the ability of the camera toaccurately render an image is essential to the performance of a trafficmonitoring system.

However, performance of a camera deteriorates over time. Thedeterioration can be gradual or sudden and can affect the camera'sability to perform its desired function. One example of decreasedperformance results when the camera captures blurred images caused by adirty or smudged lens. Deterioration can also result from cameramisorientation, which may be caused, for example, by impact to thecamera housing. Other examples of deterioration include low imagecontrast resulting from flash defects and failures and near field cameraobstruction resulting from a variety of causes.

While total failure can easily be detected by automatic means, subtledeterioration is more difficult to detect and is further complicated bythe many possible sources of deterioration including, for example,weather, vehicle speeds, and vehicle conditions. As an example,displacement of a traffic camera's field of view (FOV) can occur whenthe camera position and/or orientation is moved by a force of wind,accumulated snow/ice, and/or inadequately secured camera mounts, etc.Because traffic cameras are positioned in or near traffic flow, routineon-site camera inspections are difficult. On-site inspections candisrupt traffic and/or place technicians at risk of oncoming traffic.

Conventional approaches for detecting subtle deterioration in trafficcamera diagnostics include test pattern or scene analysis. The testpattern approach uses images with specially designed and wellcharacterized objects that are placed in the field of view of the camerato detect and/or quantify output quality by applying a process similarto one used to evaluate the quality of images rendered by multifunctionprinter devices. The test patterns are objects with known patterns (e.g.bright reflectors on a 1 ft×1 ft grid) or with readily identifiedfeatures (e.g. dark painted lines of known dimensions on a planesurface) placed on, for example, a vehicle that travels across the scenethat is being surveyed by the traffic camera. However, the labor andresources required for implementing the traffic test pattern approachare significant. Because test pattern analysis is expensive, it shouldnot be invoked unless clearly needed.

The alternative approach is scene analysis, which performs imageanalysis on all or part of the scene that is captured by the camera. Acommon shortcoming of scene-dependent image analysis is its tendency toyield lower quality and scene-dependent diagnostics signals that aresub-optimal (or even intractable) for diagnostics. This drawback can beworse if this approach is used for measuring small amounts of cameramisalignment (e.g. gradual deterioration over time). Furthermore,because scene elements and noise effects can confound diagnostic signalsthat are extracted from the image, the performance of scene analysis forcamera diagnostics varies depending on the scene.

Gaps remain and limit the capability of traffic camera diagnostics whenthese conventional approaches are used independent of one another. Amethod and system for performing traffic camera diagnostics aretherefore desired which maximize the strengths of both approaches byusing scene analysis for predicting or identifying a potentialmisalignment and then using test pattern analysis to make a finaldecision and/or correction.

BRIEF DESCRIPTION

One embodiment of the present disclosure relates to a method fordetermining a response to misalignment of a camera monitoring a desiredarea. The method includes acquiring a plurality of temporal relatedframes from the camera. One of the frames is a reference frame. Themethod further includes determining a pixel location of at least onereference object from the frames. Using the pixel location of the atleast one reference object, the method includes determining adisplacement of the camera between a current frame and the referenceframe. The method includes comparing the displacement to a firstthreshold. For the displacement exceeding the first threshold, themethod includes measuring a new displacement of the camera byintroducing at least one additional object to a camera field of view andcomparing the new displacement to a second threshold. For the newdisplacement not exceeding the second threshold, the method includesautomatically recalibrating the camera using a determined pixel locationand corresponding known physical location of the at least one additionalobject. For the new displacement exceeding the second threshold, themethod includes providing suitable notification of a detectedmisalignment to an associated user device.

Another embodiment of the present disclosure relates to a system fordetermining a response to misalignment of a camera monitoring a desiredarea. The system includes a computer having an image capture module, analignment determination module, a diagnostic response module and aprocessor adapted to process the modules. The image capture module isadapted to acquire a plurality of temporal related frames from thecamera, wherein one of the frames is a reference frame. The alignmentdetermination module is adapted to determine a pixel location of atleast one reference object from the plurality of frames. Using the pixellocation of the at least one reference object, the alignmentdetermination module determines a displacement of the camera between acurrent frame and the reference frame. The alignment determinationmodule compares the displacement to a first threshold. For thedisplacement exceeding the first threshold, the module measures a newdisplacement of the camera by introducing at least one additional objectto a camera field of view. The diagnostic response module then comparesthe new displacement to a second threshold. For the new displacement notexceeding the second threshold, the diagnostic response moduleautomatically recalibrates the camera using a determined pixel locationand corresponding known physical location of the at least one additionalobject. For the new displacement exceeding the second threshold, thediagnostic response module provides suitable notification of a detectedmisalignment to a user device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of a method for performing traffic cameradiagnostics using a multi-level approach for determining a faultpossibility and confirming the fault condition.

FIG. 2 is a schematic illustration of a traffic camera diagnostic systemin one aspect of the exemplary embodiment.

FIG. 3 is a flow chart illustrating an exemplary method for determiningand handling a traffic camera misalignment condition according toanother aspect of the exemplary embodiment.

DETAILED DESCRIPTION

The present embodiment is directed to a system and method for performingtraffic camera diagnostics using a multi-level approach including sceneand test pattern analyses. Namely, the system uses thresholding resultsof a scene analysis to determine a possible camera fault and usesthresholding results of a test pattern analysis to confirm the suspectedcamera fault.

FIG. 1 provides an overview of the method 10. The system starts adiagnostic routine at S12. At S14, a scene analysis is performed on datathat is extracted from an image frame of the scene that is captured bythe camera. The scene analysis is used to determine a severity (i.e., aprobability and/or likelihood) of the camera fault P_(S) being tested,such as, in one example, the camera misalignment and/or displacement.The subscript S is used to denote that the camera fault P_(S) isdetermined from scene analysis. The camera fault P_(S) is compared tofirst and second thresholds η₁ and η₂ at S16, S18, whereby the secondthreshold η₂ is significantly larger than the first threshold η₁. Forthe fault P_(S) not exceeding the first threshold (P_(S)≦η₁) (YES atS16), no further action is required (S20). For the fault P_(S) exceedingthe second threshold (P_(S)>η₂) (YES at S18), a maintenance crew isdispatched to physically correct the camera fault or replace the cameraat S24. This step is optional and is used to dispatch the maintenancecrew directly for the situation where the camera fault is so severe asto unequivocally be detected via scene analysis alone. For the faultP_(S) exceeding the first threshold but not exceeding the secondthreshold (η₁<P_(S)≦η₂) (NO at S16 and S18), appropriate test patternsare deployed to the field for performing more rigorous cameradiagnostics at S22 to determine severity of the camera fault P_(T). Thesubscript T is used to denote that camera fault P_(T) is determined fromtest pattern analysis. The results of the test pattern analysis are usedto confirm the suspected fault P_(S) once the initial testing hasrevealed its possibility. If the resulting severity P_(T) from the testpattern analysis S22 does not exceed a third threshold (P_(T)≦η₃) (YESat S26), then the camera is operating within specifications, and nofurther action is required, S28. If P_(T) from S22 exceeds the thirdthreshold (NO at S26), i.e., the camera fault detected from sceneanalysis is further confirmed by the test pattern analysis, thediagnostics continue to S30. At S30, the system determines whether ornot to dispatch a maintenance crew for correcting the fault. Thisdetermination is made because in some cases, such as when a camera faultis a modest displacement error, it may be possible to compensate for thefault through recalibration using the camera fault P_(T) data, and itmay not be necessary to send out the maintenance crew. If it is notpossible to perform the automatic calibration (NO at S30), then themaintenance crew is dispatched to correct the fault at S24. Althoughsome faults, such as camera misalignment, can be corrected by automaticcalibration S32, it might be desirable to maintain camera conditionsclose to an initial installation state. Therefore, it may be useful tocompare P_(T) to a fourth threshold η₄ at S31. If the camera fault P_(T)does not exceed the fourth threshold η₄ (NO at S31), the processcontinues to S32 for correcting the identified camera fault viacalibration. If the camera fault P_(T) exceeds the fourth threshold η₄(YES at S31), the change in the camera condition is too large comparedto the condition in the reference frame. The process continues to S24 todispatch the maintenance crew to correct the camera fault. The methodends at S34.

Because camera misalignment is a common fault and can well utilize theproposed method, the following more detailed description of thediagnostic method focuses on this specific camera fault. It is howeverunderstood that other camera faults, such as out-of-focus or near-fieldobstruction of the camera (e.g. due to snow, fog etc.), can be appliedas well.

FIG. 2 is a functional block diagram of a traffic camera diagnosticsystem 100 in one exemplary embodiment. The system 100 may include adeterioration determination system 102, hosted by a computing device104, such as a server computer at the service provider site, and a userdevice 106, hosted by a computing device at a customer site, such as aserver, which are linked together by communication links 108, referredto herein as a network. These components are described in greater detailbelow.

The fault determination system 102 illustrated in FIG. 2 includes aprocessor 110, which controls the overall operation of the faultdetermination system 102 by execution of processing instructions, whichare stored in memory 112 connected to the processor 110.

The camera diagnostic process disclosed herein is performed by theprocessor 110 according to the instructions stored in the memory 112. Inparticular, the memory 112 stores image capture module 114, alignmentdetermination module 116, and diagnostic response module 118.

The illustrated image capture module 114 acquires a plurality oftemporal related frames from a camera 120.

The alignment determination module 116 performs a first phase of thediagnostic routine. More specifically, the alignment determinationmodule 116 performs a scene analysis to detect a potential fault. Themodule 116 determines a pixel location of at least one reference objectfrom the plurality of frames. Using the pixel location of a referenceobject, the alignment determination module 116 determines a displacementof the object between a current frame and the reference frame. Themodule 116 compares the displacement to a threshold. For thedisplacement not exceeding the threshold, the alignment determinationmodule 116 determines an updated displacement of the object after apredetermined number of frames. For the displacement exceeding thethreshold, the alignment determination module 116 transmits thedisplacement information to the diagnostic response module 118.

The diagnostic response module 118 performs a second phase of thediagnostic routine. More specifically, the diagnostic response module118 performs a traffic test pattern analysis to confirm the suspectedfault. The module 118 measures a new displacement of the camera byintroducing at least one additional object to a camera field of view.More specifically, in one embodiment, a plurality of frames is acquiredwhile a test pattern is present in the camera FOV, and test patternanalysis is performed to determine a new displacement of the camera,preferably in terms of real-world distance units rather than pixelunits. The module 118 compares the new displacement to a secondthreshold. For the new displacement neither meeting nor exceeding thesecond threshold, the diagnostic response module 118 automaticallyrecalibrates the camera using a determined pixel location andcorresponding known physical location of the additional object, i.e.test pattern mentioned above. For the new displacement exceeding thesecond threshold, the module 118 provides suitable notification of adetected misalignment to the user device 106.

The fault determination system 102 also includes one or morecommunication interfaces (I/O), such as network interfaces 122 forcommunicating with external devices, such as the user device 106. Thevarious hardware components 110, 112, 122 of the fault determinationsystem 102 may all be connected by a bus 124.

With continued reference to FIG. 2, the fault determination system 102is communicatively linked to a user interface device (GUI) 126 via awired and/or wireless link. In various embodiments, the user interfacedevice 126 may include one or more of a display device, for displayinginformation to users, such as a notification of camera misalignment to ahuman operator 128 for review, and a user input device, such as akeyboard or touch or writable screen, for inputting instructions, and/ora cursor control device, such as a mouse, trackball, or the like, forcommunicating user input information and command selections to theprocessor 110. Specifically, the user interface device 126 includes atleast one of an input device and an output device, both of which includehardware, and which are communicatively linked with the server 104 viawired and/or wireless link(s).

With continued reference to FIG. 2, the traffic camera diagnostic system100 includes a storage device 130 that is part of or in communicationwith the deterioration determination system 102. In one embodiment, thefault determination system 102 can be in communication with a server(not shown) that hosts storage device 130, for storing a referenceobjects and locations database 132.

While the computing device 104 may be linked to as few as one camera120, in general, it can be linked to a plurality of cameras. The camera120 is not limited to any one type of camera. Rather, still cameras andvideo cameras are contemplated for surveying a desired region. Thecamera 120 is adapted to capture a plurality of image frames andtransmit the image and/or video data to the deterioration determinationsystem 102. In the contemplated embodiment, the camera 120 can be usedfor speed enforcement applications, but the purpose of the surveillanceis not limited to any one application.

The memory 112, 130 may represent any type of tangible computer readablemedium such as random access memory (RAM), read only memory (ROM),magnetic disk or tape, optical disk, flash memory, or holographicmemory. In one embodiment, the memory 112, 130 may each comprise acombination of random access memory and read only memory. The digitalprocessor 110 can be variously embodied, such as by a single-coreprocessor, a dual-core processor (or more generally by a multiple-coreprocessor), a digital processor and cooperating math coprocessor, adigital controller, or the like. The digital processor 110, in additionto controlling the operation of the respective deteriorationdetermination system 102, executes instructions stored in memory 112,130 for performing the parts of the method outlined below.

The software modules as used herein, are intended to encompass anycollection or set of instructions executable by the fault determinationsystem 102 so as to configure the computer or other digital system toperform the task that is the intent of the software. The term “software”as used herein is intended to encompass such instructions stored instorage medium such as RAM, a hard disk, optical disk, or so forth, andis also intended to encompass so-called “firmware” that is softwarestored on a ROM or so forth. Such software may be organized in variousways, and may include software components organized as libraries,Internet-based programs stored on a remote server or so forth, sourcecode, interpretive code, object code, directly executable code, and soforth. It is contemplated that the software may invoke system-level codeor calls to other software residing on the server or other location toperform certain functions.

The communication interfaces 122 may include, for example, a modem, arouter, a cable, and and/or Ethernet port, etc.

As will be appreciated, while two computing devices 104, 106 areillustrated by way of example, the system 100 may be hosted by fewer ormore linked computing devices. Each computing device may include, forexample, a server computer, desktop, laptop, or tablet computer,smartphone or any other computing device capable of implementing themethod described herein.

FIG. 3 is a flow chart illustrating an exemplary method 300 fordetermining and handling a traffic camera fault condition according toanother aspect of the exemplary embodiment. While a diagnostic routinecan fail outright when a fault is large and/or significant enough,gradual faults can be less readily apparent, even if image framesundergo visual inspection. Measurement errors can increase significantlyand without warning in gradual faults. Therefore, the exemplary methodis proposed to identify and correct for gradual camera deterioration andfaults in camera misalignment, in particular.

The method starts at S302. During a course of the camera's use, thenatural scene (i.e., images that the camera acquires as part of itsregular use) is periodically analyzed for determining potentialdisplacement in the camera field of view. Accordingly, the image capturemodule 114 acquires a plurality of temporal related frames from thecamera 120 at S304. In the contemplated embodiment, the image framesbelong to a video that the camera captures at the scene. In otherembodiments, however, the image frames can include a series of stillshots captured by the camera in a short period.

The method first determines whether a potential camera displacementexists based on an analysis of the natural scene. In other words,diagnostic signals can be extracted from the scene directly. The systeminstitutes a scene analysis in a first diagnostic phase at S305.Depending on the results, the system can implement further diagnosticactions and/or correct the fault. The analysis of the natural scene isperformed by monitoring changes of estimated vanishing points oridentified key-points (hereinafter referred to as ‘objects’) of the roadscene from the acquired image frames. Generally, the changes aremonitored between a reference frame and a current frame.

Therefore, the alignment determination module 116 determines if thecurrent frame is a reference frame in a sequence at S306. A referenceframe is defined herein as including a frame captured at a time instantwhen the camera 120 is free of misalignment error. The analysis (e.g.estimating the positions of vanishing points or identified key-points)may be performed on a plurality of frames over a substantially shortperiod of time; and the average or other statistical measure, such as,median, min, max etc., of the results may be used as the reference. Thisapproach is preferable to using the analysis of a single referenceframe, since the camera system has noise.

For simplicity, the terms reference frame and/or current frame are usedthroughout the rest of the disclosure. It should be understood that areference frame may include the analysis of a series of referenceframes, which includes a plurality of frames over a substantially shortperiod of time that is close to the reference time. Similarly, a currentframe may include the analysis of a series of current frames, whichincludes a plurality of frames over a substantially short period of timethat is close to the said current time.

For the current frame being the reference frame in the sequence (YES atS306), the system determines a pixel location of at least one referenceobject in the reference frame at S308. A reference object is definedherein as a selected fixed object or vanishing point in the scene. Thereference object can include a stationary object and, more specifically,a stationary and permanent object, within the camera field of view.Non-limiting examples of objects can include a traffic indicator, suchas a traffic light or sign, a lamppost, a building, a tollbooth, aroadway lane marker, and a combination of the above. The object that isselected for the scene analyses portion of the diagnostics routine canvary from site to site depending on the intended surveillance functionof the camera. For example, for red light enforcement, a common sceneelement, such as the traffic light, can be selected as the object usedin detecting camera displacement. However, noises, such as shaking ofthe traffic light, can result in detection errors. Because the impact ofmild camera displacement on system performance is not large, the redlight can still be used to detect gross displacement in a stopenforcement application.

However, a small camera displacement can result in a significantmeasured speed error for a camera that is used for speed measurementand/or enforcement. Slight camera displacement can upset thepredetermined spatial calibration (pixels to real-world coordinates) ofthe camera. Furthermore, for a camera that is programmed to captureimages when objects move across select pixel locations, the displacementaffects the objects' locations when the images are captured. In a casewhere the object is a vehicle, the slight displacement can affect theoptimal location for license plate reading. Therefore, there is nolimitation made herein to the object selected. In one embodiment, thereference object can include vanishing points that are derived fromimage analysis of the frame. Detecting vanishing points from imageanalysis of an arbitrary scene is well-known in image processingtechnology and prior arts. For example, vanishing points can beestimated by finding the converging points of any two parallel lines oredges in the scene, such as, e.g., lane divider markers or rectangularroad signs. Other examples, such as using general statistics of edgeorientation in the scene, can be used for determining vanishing points.This reference object and corresponding location can be stored in adatabase 132 and accessed for the processing of subsequent frames.

For the current frame not being the reference frame in a sequence (NO atS306), the module determines at S310 a pixel location of the referenceobject in the current frame. In the contemplated embodiment, the currentframe is analyzed to detect the reference object. Any process understoodin the art can be used for locating the reference object in the currentframe. The pixel location is determined where the detected referenceobject appears in the frame.

The identified pixel locations of the reference object in the referenceframe and the current frame respectively are used in the first phaseS305 of the diagnostic routine. In the contemplated embodiment, thealignment determination module 116 determines a displacement of thecamera between the current frame and the reference frame at S312.However, other fault conditions of the camera can be measured bycomparing different characteristics of the reference object between thereference and current frames. For example, in the case of camera that isout of focus, instead of using a comparison of pixel locations of thereference object, other characteristics can be used, such as, thesharpness (e.g. measured by average strength of edges) and/or scaling ofthe object size etc., for detecting whether the camera focus has changedover time.

For embodiments that use at least two reference objects for determiningthe camera displacement, a displacement is computed for each referenceobject. An average displacement is then computed for all the referenceobjects. Note that a use of at least two reference objects can be anycombination of (i) using more than one reference object in a frame and(ii) using one reference object in a plurality of frames acquired over asubstantially short period of time.

Continuing with FIG. 3, the computed displacement is compared to a firstpredetermined threshold T₁ at S314. Generally, the results of thecomparison are used to rule out a fault in need of remediation. Morespecifically, for a displacement not exceeding the threshold value T₁(NO at S314), the system recomputes the displacement after apredetermined number m of frames at S316. After recomputing thedisplacement after the m^(th) frame, the process returns to S314 and thealignment determination module 116 compares the updated displacementwith the threshold T₁. A displacement exceeding the threshold value T₁(YES at S314) is indicative of the potential misalignment of the camera.In other words, the camera has potentially shifted since it was lastcalibrated. To confirm the determination, the system institutes thetraffic test pattern analysis in a second diagnostic phase at S317.

However, because a potential fault that is detected from the naturalscene can result from a noisy measurement, the first phase analysis atS305 can result in false readings. Therefore, an embodiment iscontemplated as including an intermediate determination by comparing thedisplacement measurement in the scene analysis to a threshold κ1 asshown in S16 in FIG. 1.

In one embodiment, the threshold value T₁ can be based on a relationshipbetween a measured parameter and a final output parameter. For example,in an embodiment including a speed enforcement traffic camera 120, thepredetermined threshold T₁ can be based on a relationship between cameradisplacement and speed error. An allowable speed error limit can be usedto find a corresponding displacement error threshold. In preferredembodiments, the predetermined threshold T₁ can consider a confidence inthe accuracy of the scene analysis results, an impact of the resultingcamera fault on the purpose of the traffic surveillance camera, and acost of implementing the traffic test pattern analysis in the seconddiagnostic phase S317.

The traffic test pattern analysis of S317 differs from the sceneanalysis of S305 because it processes a test pattern that is insertedinto the scene, rather than reference objects that happen to occur inthe scene. One advantage of using test patterns includes better controlof the reference objects (easier to identify, better placements in theFOV of the camera, etc.). Another advantage is that the diagnosticsignals are scene independent (since test patterns can be unified tostandard patterns and thus are not scene dependent), etc. More detailedadvantages of test-pattern analysis can be found in U.S. Ser. No.13/371,068, filed Feb. 10, 2012, titled “Traffic Camera Diagnostics viaTest Targets”, by Wencheng Wu et al, the contents of which are entirelyincorporated herein. This phase S317 can provide for greater accuracy ofmeasuring displacement because specially designed test patterns andcorresponding analyses result in better diagnostic signals without beinglimited to what the scene can offer. For diagnosing traffic cameradisplacement, a test pattern based method can be implemented asdescribed in the '068 disclosure.

The second phase S317 starts by introducing at least one additionalobject to the camera field of view. In one embodiment, the additionalobject is a test target with predetermined reference position markers.In one embodiment, the test target, such as a set of bright reflectorson a 1 ft×1 ft grid, can be placed on a moving object that carries thetarget across the scene. In this manner, the position of the targetchanges from frame to frame. In another embodiment, the additionalobject can include a stationary object of known location beingintroduced into the camera field of view. Non-limiting examples ofstationary objects can include a sign, a traffic cone, a man-madeobject, and a combination of above. Namely, the object is not limited toany one object, but can include any object easily identified by imageanalysis.

The selected test pattern can be based on the type of camera fault thatis being diagnosed. The information provided by the scene analysis canbe used as a guide for the design of the test-pattern. In one example, aladder chart can be used for a suspected blur-based or out-of-focus typefault. In another example, a grid of reflectors can be used for a likelycamera displacement. Accordingly, one aspect of performing the sceneanalysis in the first phase S305 is that the results enable a smaller,and more custom, set of test patterns to be deployed in the second phaseS317.

The diagnostic response module 118 measures a new displacement at S318by analyzing images/frames, which are acquired during the time when theinserted additional test patterns/objects are present in the scene. Thatis, S318 only operates when the inserted test patterns/objects aretemporarily present in the scene. Because an insertion of testpatterns/objects implies a cost (such as the cost of sending a vehiclecarrying the test target or the cost of interrupting traffic totemporarily place the stationary objects), the second phase S317preferably is executed only when a potential fault is detected from thenatural scene S305. The new computed displacement is compared to apredetermined threshold T₂ at S320. In the case of camera misalignment,S26 can be omitted in FIG. 1 since it is desired to correct themisalignment via camera re-calibration if possible.

Continuing with FIG. 3, for a displacement not exceeding the thresholdvalue T₂ (NO at S320), the diagnostic response module 118 automaticallyrecalibrates the camera using the additional object at S322. Mild cameramisalignment can be remedied without sending a maintenance crew. Forexample, the collected test pattern data can be used to derive anupdated camera calibration as discussed in U.S. Ser. No. 13/527,673,titled “Camera Calibration Application” by Martin H. Hoover et al., thecontents of which are incorporated herein in their entirety. One aspectof a calibration response resulting in this phase S317 is that the extracost of dispatching a maintenance crew is not incurred. The module 118computes a determined pixel location and corresponding known physicallocation of the additional object at S322. Accordingly, the pixellocation and the additional object are stored in the database 132. Achange of the camera's field of view can be incorporated into the speedmeasurement by updating the calibration.

Continuing with FIG. 3, a maintenance crew may need to reposition thecamera for a larger displacement or for instances when calibrationcannot remedy the problem, such as, for example, when the misalignmentis severe, when faults are caused by near-field blockage, duringout-of-focus conditions, and/or when there is illuminator failure, etc.A displacement exceeding the threshold value T₂ (YES at S320) confirmsthat the camera has incurred a serious fault and that the camera is inneed of on-site repair, maintenance, and/or replacement. The diagnosticresponse module 118 provides suitable notification of the detected faultat S326 to a user device 106. The notification can include aninstruction for the maintenance crew to visit the camera site. Infurther contemplated embodiments, the notification can indicate a typeand/or cause of the fault, and it can indicate and/or suggest thenecessary (e.g., replacement) parts that the crew need to bring to thecamera site.

Because this response, i.e., sending a maintenance crew, can be the mostexpensive in a series of options, it is the last resort for manyapplications. Therefore, one aspect of the discussed process is tominimize false alarms which can needlessly initiate this response.Additionally, the diagnostic information that is provided by earliersteps (S305) can reduce the cost for this response (S326), such as costsassociated with frequently deploying a test target and with sendingmaintenance crews to revisit sites.

The process continues by determining if the current frame is the lastframe in the acquired sequence at S328. For the current frame not beingthe last frame (NO at S328), the system sets the next frame as thecurrent frame at S330 and returns to S310 for processing the next frame.For the current frame being the last frame in the sequence (YES atS328), the method ends at S332. Note that it is often unnecessary toanalyze each frame acquired by the camera for the purpose ofdiagnostics. Though it is not explicitly stated, it should be understoodthat the term “current” frame may mean the frames that are currentlybeing analyzed for camera fault diagnostics, which may be requested bythe system at a regular schedule (such as every hour during the day andevery 4 hours at night) or an irregular schedule (such as one week afterthe last camera maintenance) or a combination of both.

Although the present disclosure uses a test pattern to improvediagnostic signals provided by a scene analysis diagnostic approach,other embodiments are contemplated to use a test pattern for providingauxiliary or complementary diagnostic signals. For example, testpatterns can be used to help a scene analyzer understand the scene (e.g.road geometry and travel directions) in addition to the earlierdiscussed aspect of providing more precise measurements of certaindiagnostic signals. In one embodiment, the scene analysis can be used todetermine a cause of a fault while the test pattern analysis can be usedto determine whether the fault is relevant to the purpose of the camera.

The underlying concept for this disclosure is to use a lower cost sceneanalysis approach as a first level of defense for camera diagnostics andto use a more accurate test pattern approach for confirmation whennecessary.

Although the method in FIGS. 1 and 3 are illustrated and described abovein the form of a series of acts or events, it will be appreciated thatthe various methods or processes of the present disclosure are notlimited by the illustrated ordering of such acts or events. In thisregard, except as specifically provided hereinafter, some acts or eventsmay occur in different order and/or concurrently with other acts orevents apart from those illustrated and described herein. It is furthernoted that not all illustrated steps may be required to implement aprocess or method in accordance with the present disclosure, and one ormore such acts may be combined. The illustrated methods and othermethods of the disclosure may be implemented in hardware, software, orcombinations thereof, in order to provide the control functionalitydescribed herein, and may be employed in any system including but notlimited to the above illustrated system 100, wherein the disclosure isnot limited to the specific applications and embodiments illustrated anddescribed herein.

The method illustrated in FIGS. 1 and 3 may be implemented in a computerprogram product that may be executed on a computer. The computer programproduct may comprise a non-transitory computer-readable recording mediumon which a control program is recorded (stored), such as a disk, harddrive, or the like. Common forms of non-transitory computer-readablemedia include, for example, floppy disks, flexible disks, hard disks,magnetic tape, or any other magnetic storage medium, CD-ROM, DVD, or anyother optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, or othermemory chip or cartridge, or any other tangible medium from which acomputer can read and use.

Alternatively, the method may be implemented in transitory media, suchas a transmittable carrier wave in which the control program is embodiedas a data signal using transmission media, such as acoustic or lightwaves, such as those generated during radio wave and infrared datacommunications, and the like.

The exemplary method may be implemented on one or more general purposecomputers, special purpose computer(s), a programmed microprocessor ormicrocontroller and peripheral integrated circuit elements, an ASIC orother integrated circuit, a digital signal processor, a hardwiredelectronic or logic circuit such as a discrete element circuit, aprogrammable logic device such as a PLD, PLA, FPGA, Graphical card CPU(GPU), or PAL, or the like. In general, any device, capable ofimplementing a finite state machine that is in turn capable ofimplementing the flowchart shown in FIGS. 1 and 3 can be used toimplement the method.

It will be appreciated that variants of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intomany other different systems or applications. Various presentlyunforeseen or unanticipated alternatives, modifications, variations orimprovements therein may be subsequently made by those skilled in theart which are also intended to be encompassed by the following claims.

What is claimed is:
 1. A method for determining a response tomisalignment of a camera monitoring a desired area, said methodcomprising: acquiring a plurality of temporal related frames from saidcamera, one of said frames being a reference frame; determining a pixellocation of at least one reference object in said reference frame and acurrent frame; determining a displacement of said camera by comparingsaid pixel location of said at least one reference obiect in saidcurrent frame and said reference frame; comparing said displacement to afirst displacement error threshold; for said displacement not exceedingsaid first displacement error threshold, determining an updateddisplacement of said camera after a predetermined number of frames andcomparing said updated displacement to said first threshold; for saiddisplacement exceeding said first displacement error threshold,measuring a new displacement of said camera by introducing at least oneadditional object to a camera field of view, and comparing said newdisplacement to a second displacement error threshold; for said newdisplacement not exceeding said second displacement error threshold,automatically recalibrating said camera using a determined pixellocation and corresponding known physical location of said at least oneadditional object; and, for said new displacement exceeding said seconddisplacement error threshold, providing suitable notification of adetected misalignment to an associated user device.
 2. The method ofclaim 1, wherein said at least one reference object is selected from agroup consisting of: stationary objects within said camera field ofview; and, vanishing points derived from image analysis of said frame.3. The method of claim 1, wherein determining said pixel location of atleast one reference object includes: analyzing said frame to detect saidobject appearing in said frame; and, determining said pixel locationwhere said detected object appears in said frame.
 4. The method of claim1, wherein for said frames including at least two reference objects,determining said camera displacement comprises: computing a displacementfor each reference object; and, computing an average displacement forsaid at least two reference objects.
 5. The method of claim 1, whereinsaid additional object is one of a test target with predeterminedreference position markers, present in or moving through said area and astationary object of known location being situated within said camerafield of view and being selected from a group consisting of: a sign, atraffic cone, any man-made objects that can be easily identified byimage analysis; and, a combination of said above.
 6. The method of claim2, wherein said stationary object is situated within said camera fieldof view and is a defined part of an object selected from a groupconsisting of: a traffic signal light, a traffic sign, a lamppost, abuilding, a tollbooth, a roadway lane marker, and a combination of saidabove.
 7. An apparatus that executes said method of claim
 1. 8. Anon-transitory machine-readable medium including a computer programwhich when executed performs said method of claim
 1. 9. A system fordetermining a response to misalignment of a camera monitoring a desiredarea, said system comprising a computer device including a memory incommunication with a processor configured to: acquire a plurality oftemporal related frames from said camera, one of said frames being areference frame; determine a pixel location of at least one referenceobject from said frames in said reference frame and a current frame;determine a displacement of said camera by comparing said pixel locationof said at least one reference object in said current frame and saidreference frame; compare said displacement to a first displacement errorthreshold; for said displacement not exceeding said first displacementerror threshold, determine an updated displacement of said camera aftera predetermined number of frames and compare said updated displacementto said first displacement error threshold; for said displacementexceeding said first displacement error threshold, measure a newdisplacement of said camera by introducing at least one additionalobject to a camera field of view, and compare said new displacement to asecond displacement error threshold; for said new displacement notexceeding said second displacement error threshold, automaticallyrecalibrate said camera using a determined pixel location andcorresponding known physical location of said at least one additionalobject; and, for said new displacement exceeding said seconddisplacement error threshold, provide suitable notification of adetected misalignment to an associated user device.
 10. The system ofclaim 9, wherein said at least one reference object is selected from agroup consisting of: a stationary object within said camera field ofview; and, vanishing points derived from image analysis of said frame.11. The system of claim 9, wherein said processor is further configuredto: analyze said current frame to detect said object appearing in saidcurrent frame; and, determine said pixel location where said detectedobject appears in said current frame.
 12. The system of claim 9, whereinfor said frames including at least two reference objects, said processoris further configured to: compute a displacement for each referenceobject; and, compute an average displacement for said at least tworeference objects.
 13. The system of claim 9, wherein said additionalobject is one of a test target with predetermined reference positionmarkers, present in or moving through said area and a stationary objectof known location being situated within said camera field of view andbeing selected from a group consisting of: a sign, a traffic cone, anyman-made objects that can be easily identified by image analysis; and, acombination of said above.
 14. The system of claim 10, wherein saidstationary object is situated within said camera field of view and is adefined part of an object selected from a group consisting of: a trafficsignal light, a traffic sign, a lamppost, a building, a tollbooth, aroadway lane marker, and a combination of said above.
 15. The system ofclaim 9 further comprising: an image capture device in communicationwith said computer device and being adapted to capture said plurality offrames and transmit pixel data describing at least one select frame tosaid image capture module.
 16. The system of claim 9 further comprising:a user device in communication with said computer device, said userdevice including a graphical user interface for outputting saidnotification to a user.
 17. The system of claim 9 further comprising: astorage device in communication with said computer device and beingadapted to store said pixel location of at least one reference object insaid reference frame.